Automated sign language translation and communication using multiple input and output modalities

ABSTRACT

Methods, apparatus and systems for recognizing sign language movements using multiple input and output modalities. One example method includes capturing a movement associated with the sign language using a set of visual sensing devices, the set of visual sensing devices comprising multiple apertures oriented with respect to the subject to receive optical signals corresponding to the movement from multiple angles, generating digital information corresponding to the movement based on the optical signals from the multiple angles, collecting depth information corresponding to the movement in one or more planes perpendicular to an image plane captured by the set of visual sensing devices, producing a reduced set of digital information by removing at least some of the digital information based on the depth information, generating a composite digital representation by aligning at least a portion of the reduced set of digital information, and recognizing the movement based on the composite digital representation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent document is a continuation of U.S. patent application Ser.No. 17/208,968, filed Mar. 22, 2021, and entitled “AUTOMATED SIGNLANGUAGE TRANSLATION AND COMMUNICATION USING MULTIPLE INPUT AND OUTPUTMODALITIES,” which is a continuation of U.S. patent application Ser. No.16/694,965, filed Nov. 25, 2019, and entitled “AUTOMATED SIGN LANGUAGETRANSLATION AND COMMUNICATION USING MULTIPLE INPUT AND OUTPUTMODALITIES”, which is a continuation of U.S. patent application Ser. No.16/258,509, filed Jan. 25, 2019, and entitled “AUTOMATED SIGN LANGUAGETRANSLATION AND COMMUNICATION USING MULTIPLE INPUT AND OUTPUTMODALITIES”, which claims priority to and the benefits of U.S.Provisional Patent Application No. 62/664,883 filed on Apr. 30, 2018,entitled “MULTI-APERTURE SIGN LANGUAGE RECOGNITION METHOD ANDAPPARATUS”, and U.S. Provisional Patent Application No. 62/629,398,filed Feb. 12, 2018, entitled “INTERACTIVE AUTOMATED SIGN LANGUAGETRANSLATION METHOD AND APPARATUS”. The entire contents of thebefore-mentioned patent applications are incorporated by reference aspart of the disclosure of this patent document.

TECHNICAL FIELD

This document generally relates devices to enable communications, andmore particularly to using multiple modalities for communication thatinclude patterns or gestures.

BACKGROUND

Machine assisted interpersonal communication has simplified bothbusiness and personal communications, and has enabled the source andreceiver of a communication to be separated in both time and space.Devices for machine assisted interpersonal communication range from thesimple answering machine to smartphone-based translation systems thatcan interpret a language (e.g., French) and translate it into anotherlanguage for the smartphone user (e.g., spoken or written English).

One specific application of machine assisted interpersonal communicationis sign language translation. A sign language (also known as signedlanguage) is a language that uses manual communication to conveymeaning, ideas and thoughts, which simultaneously employs hand gestures,movement, orientation of the fingers, arms or body, and facialexpressions to convey a speaker's ideas. The complexity of sign languagemay be captured, in part, by using multiple input and output modalitiesfor its translation and communication.

SUMMARY

Disclosed are devices, systems and methods for using multiple input andoutput modalities that can be used to capture and process images forvarious applications, including automated sign language translation andcommunication.

In one aspect, the disclosed technology may be used to recognize a signlanguage communicated by a subject. This method includes capturing atleast one movement associated with the sign language using a set ofvisual sensing devices, where the set of visual sensing devices includesmultiple apertures oriented with respect to the subject to receiveoptical signals corresponding to the at least one movement from multipleangles. The method further includes generating digital informationcorresponding to the at least one movement based on the optical signalsfrom the multiple angles, and collecting depth information correspondingto the at least one movement in one or more planes perpendicular to animage plane captured by the set of visual sensing devices. The abovemethod additionally includes producing a reduced set of digitalinformation by removing at least some of the digital information basedon the depth information, generating a composite digital representationby aligning at least a portion of the reduced set of digitalinformation, and recognizing the at least one movement based on thecomposite digital representation.

In another aspect, the disclosed technology may be used to recognize asign language communicated by a subject. This method includes capturingat least one hand gesture associated with a movement in the signlanguage using a set of visual sensing devices, the set of visualsensing devices comprising multiple apertures oriented with respect tothe subject to receive optical signals corresponding to the at least onemovement from multiple angles, generating digital informationcorresponding to the at least one hand gesture based on the opticalsignals from the multiple angles, capturing at least one environmentalfactor using a set of non-visual sensing devices, combining the digitalinformation with information associated with the at least oneenvironmental factor to improve the recognition of the movement in thesign language.

In yet another aspect, the disclosed technology may be used to recognizea sign language communicated by a subject. This method includescapturing at least one movement associated with the sign language usinga set of visual sensing devices, the set of visual sensing devicescomprising multiple apertures oriented with respect to the subject toreceive optical signals corresponding to the at least one movement frommultiple angles, generating digital information corresponding to the atleast one movement based on the optical signals from the multipleangles, and recognizing the at least one movement based on the digitalinformation.

In yet another aspect, an apparatus comprising a memory and a processorimplements the above-described methods is disclosed.

In yet another aspect, the method may be embodied asprocessor-executable code and may be stored on a non-transitorycomputer-readable program medium.

The above and other aspects and features of the disclosed technology aredescribed in greater detail in the drawings, the description and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a two-way translation system used by two parties inaccordance with an example embodiment of the disclosed technology.

FIG. 2 illustrates a remote two-way translation system used by twoparties that may be in different locations over a communication networkin accordance with an example embodiment of the disclosed technology.

FIG. 3 illustrates a one-way translation system used by two parties inaccordance with an example embodiment of the disclosed technology.

FIG. 4 illustrates another two-way interactive translation systemimplemented to enable communications by two parties in accordance withan example embodiment of the disclosed technology.

FIG. 5 illustrates a configurable automated translation system inaccordance with an example embodiment of the disclosed technology.

FIG. 6 illustrates another configurable automated translation system inaccordance with an example embodiment of the disclosed technology.

FIG. 7 illustrates yet another configurable automated translation systemin accordance with an example embodiment of the disclosed technology.

FIG. 8A illustrates one view of an image capture and processing devicethat can be used for automated sign language translation in accordancewith an example embodiment of the disclosed technology.

FIG. 8B illustrates another view of an image capture and processingdevice that can be used for automated sign language translation inaccordance with an example embodiment of the disclosed technology.

FIG. 9 illustrates a flow diagram of operations that can be carried outby various component to implement automated sign language translation inaccordance with an example embodiment of the disclosed technology.

FIG. 10 illustrates a method that includes a set of operations that canbe carried out to automate sign language translation in accordance withan example embodiment of the disclosed technology.

FIG. 11 illustrates an example system for sign language recognitionusing a device with multiple input and output modalities.

FIG. 12 illustrates another example system for sign language recognitionusing a device with multiple input and output modalities.

FIGS. 13A, 13B and 13C illustrate an example device for sign languagerecognition using a device with multiple input and output modalities.

FIG. 14 illustrates example components of a system using a device forsign language recognition using a device with multiple input and outputmodalities.

FIG. 15 illustrates a flowchart of an example method for sign languagerecognition using a device with multiple input and output modalities.

FIG. 16 illustrates a flowchart of another example method for signlanguage recognition using a device with multiple input and outputmodalities.

FIG. 17 illustrates a flowchart of yet another example method for signlanguage recognition using a device with multiple input and outputmodalities.

DETAILED DESCRIPTION

Machine-assisted interpersonal communication (or technology-assistedcommunication) involves one or more people communicating by means of amechanical or electronic device or devices with one or more receivers.The devices that are used can give the communication permanence (e.g.,storage devices) and/or extend its range (e.g., wireless communication)such that the source and receiver can be separated in time and space.

One specific application of using devices for machine-assistedinterpersonal communication is sign language communication andtranslation. Sign languages are extremely complex, and generally do nothave a linguistic relation to the spoken languages of the lands in whichthey arise. The correlation between sign and spoken languages is complexand varies depending on the country more than the spoken language. Forexample, the US, Canada, UK, Australia and New Zealand all have Englishas their dominant language, but American Sign Language (ASL), used inthe US and English-speaking Canada, is derived from French Sign Languagewhereas the other three countries sign dialects of British, Australian,and New Zealand Sign Language (collectively referred to as BANZSL).Similarly, the sign languages of Spain and Mexico are very different,despite Spanish being the national language in each country.

Furthermore, unlike spoken languages, in which grammar is expressedthrough sound-based signifiers for tense, aspect, mood, and syntax, signlanguages use hand movements, sign order, and body and facial cues tocreate grammar. In some cases, even certain uttered sounds or clicks mayform a part of the sign language. Such a cue is referred to as anon-manual activity and can vary significantly across different signlanguages. It is desirable for a sign-language translation system tocapture and process both the hand movements and the non-manualactivities to provide an accurate and natural translation for theparties.

Embodiments of the disclosed technology that are implemented for signlanguage translation are flexible and adaptable in that an input signlanguage, which can be any one of a several sign languages, is convertedto an internal representation, which can then be used to translate theinput sign language into one or more of a variety of output signlanguages. Furthermore, the embodiments described in this documentemploy a multiplicity of different sensors and processing mechanisms tobe able to capture and process information that may not be obtainablewhen a single sensor or process is utilized, and to facilitate accuratecapture, processing and interpretation of the information to allowtranslation between different sign languages. In an example, the Biblemay be translated from any language to a particular sign language, orfrom one sign language representation to another, based on theembodiments disclosed in this document. In general, any textual, audibleor sign language content may be translated in real-time to correspondingcontent in another audible, textual or sign language.

FIGS. 1-10 are illustrations offered to provide the proper context forthe specific application of a sign language translation system that canbenefit from the training techniques described in later sections of thisdocument. FIG. 1 illustrates a two-way translation system used by twoparties in accordance with an example embodiment of the disclosedtechnology. As illustrated in FIG. 1 , a device 110 facilitatescommunication between a first party 101 and a second party 102. Thedevice 110 comprises two sets of sensor inputs and outputs for each ofthe users. In an example, an outgoing communication of the first party(who may be a sign language user) may be a visual language, a facialexpression, or a textual language or input. The device 110 identifiesthe language used by the first party and translates it into a languageunderstandable by the second party, and outputs it based on a preferenceof the second party. In another example, as a part of the incomingcommunication, the device may provide the translated output as a visuallanguage (e.g. another sign language) that may include glyphs,animations or video synthesis (e.g. avatars), or in an audible ortextual language.

This process can be inverted by the device in that an outgoingcommunication of the second party, which now may also be in an audiblelanguage, is identified and translated for the first party. The devicemay output the translation as an incoming communication for the party asa type of visual language or a textual language. The device may inputthe visual language, audible language, facial expression, or texturallanguage or input as an outgoing communication from the party. In someembodiments, the language choice or preference of either party may beidentified by the device. In other embodiments, the language choice orpreference may be predetermined or selected in real-time. It is notedthat the example system of FIG. 1 allows communications between two signlanguage users, or a sign language user and a non-sign language user.

FIG. 2 illustrates a remote two-way translation system used by twoparties that may be in different locations over a communication networkin accordance with an example embodiment of the disclosed technology. Asillustrated in FIG. 2 , the first party 201 and a second party 202 neednot necessarily be co-located as long as they have access to acommunication network that allows the exchange of information from onelocation to another location. In the depicted scenario, two devices 210and 220 are connected via a communication network, which can be a wirednetwork or a wireless network such as a Wi-Fi network, a personal areanetwork, or a mobile network. As in the case of FIG. 1 , the remotetwo-way translation system allows communications between two signlanguage users, or a sign language user and a non-sign language user.

FIG. 3 illustrates a one-way translation system used by two parties 301,302 in accordance with an example embodiment of the disclosedtechnology. This example includes some features and/or components thatare similar to those illustrated in FIGS. 1-2 , and described above, andtheir description is not repeated. As illustrated in FIG. 3 , one ormore sensors 310 capture one or more aspects of the sign languagespeaker and/or the speaker's environment and generate a digitalrepresentation of what is being observed. As will be described in latersections of this document, the one or more sensors 310 can include avariety of audio, video, motion, haptic and other types of sensors. Insome embodiments, the video rate of the sensor data capture may beselected based on the sign language input due to the increasedcomplexity of some sign languages. The digital representation of thesign language communication may include one or more gestures, facialcues, body cues, or environmental factors.

The captured information, including the captured video, is thenprocessed by one or more processors 320 to identify the input signlanguage, recognize individual gestures and other features of thecommunication, and translate the communication to an internalrepresentation. The internal representation of the sign languagecommunication can then be converted to an appropriate language and/orformat and displayed or audibly output in the language of the secondparty by various output devices 330, such as displays, speakers, andhaptic devices. In some embodiments, the second language may be either apredetermined language or selected by the second party. In otherembodiments, a second translation or transformation may be performed ifit is detected that certain output devices are not present, or if theuser selects an alternate output option.

FIG. 4 illustrates another two-way interactive translation systemimplemented to enable communications by two parties 401, 402 inaccordance with an example embodiment of the disclosed technology. Asillustrated in FIG. 4 , the translation system includes one or moresensors 410, one or more processors 420, and various output devices thatare similar to the components described above, and their description isnot repeated. In FIG. 4 , the one or more sensors 410 are able toreceive audible or physical input from the second party 402, who wishesto communicate with the sign language speaker (the first party 401). Insome embodiments, the translation system includes additional inputinterfaces, such as a keyboard or a touchscreen, to receive physicalinput from the second party 402.

The audible or textual input from the second part is processed by theprocessor and converted to the internal representation. This internalrepresentation of the second party's communication is then translated tothe sign language of the first party 401 and displayed via a secondarydisplay 460. In some embodiments, the first party may receive the inputas text, graphic (glyph-like) or through an animated figurerepresentation of the second party. In other embodiments, the two-waytranslation between a sign language and a textual, audible or differentsign language may be performed in real-time.

FIG. 5 illustrates a configurable automated translation system inaccordance with an example embodiment of the disclosed technology. Asillustrated in FIG. 5 , embodiments of the disclosed technology mayinclude a number of different visual language sensors 510. In anexample, the visual language sensors may include one or more of an RGBcolor camera, a monochrome camera, a 3D stereo camera, structured lightemitter, a 3D processor of structured light, a time-of-flight emitterand camera, a non-visual electromagnetic sensor and a non-visualelectro-optical sensor. The system may also include standard inputdevices 520 [Perkins Coie will correct the drawings to include thisdevice], e.g. a microphone, a microphone array or 3D microphone, atouchscreen keyboard, or a physical keyboard.

In addition to the input sensors described above, the device includes ahost of output capabilities. For example, standard language renderingmay be performed using a textual display 540 or a speaker 530. On theother hand, the sign language output may include textual, graphical(glyphs, etc.), animated (virtual hands, avatars, etc.) or synthesizedvideo (from a library of basic visual language gestures) outputs, whichcan be demonstrated to the user via another textual display 540 orspeaker 530.

FIG. 5 also illustrates that the processing of the input language fromthe first party, and specifically the translation from an input languageto the internal representation and subsequently to the language of thesecond party, can be performed either locally, remotely or both. In someembodiments, the device may have access to cloud computing resources,which may be leveraged in, for example, configurations where manydifferent output sign languages are to be supported.

FIG. 6 illustrates another configurable automated translation system inaccordance with an example embodiment of the disclosed technology. Asillustrated in FIG. 6 , the translation system includes one or moresensors 610, one or more processors 620, and various output devices thatare similar to the components described in the examples above, and thecorresponding description is not repeated. In some embodiments, thefirst party 601 or the second party 602 is not necessarily a person butcould be automata. For example, a sign language user may communicatewith a virtual assistant, an interactive response agent, or simply analert generation mechanism. Embodiments of the disclosed technology areflexible and adaptable to be able to support the translation oflanguages between sign language users, audible language speakers, andautomata, and any combination of the above. In part, this is achieved bytranslating the input language to an internal representation, and thentranslating it to the required one or more output languages.

In an example, the Bible may be translated into American Sign Language(ASL) which is one of the most commonly used sign languages. Expertinput, e.g. interpretation and context for specific verses or sections,may be used to improve the translation during the training period. TheASL-translated Bible may be then displayed using an avatar in a lesscommonly used sign language that is not ASL. In some embodiments, boththe first and second parties may be sign language users, andfurthermore, may not use the same sign language.

FIG. 7 illustrates yet another configurable automated translation systemin accordance with an example embodiment of the disclosed technology.The automated sign language translation system can be used to translatespecific literature or material, e.g. the Bible or works by a particularauthor. In these scenarios, a remote expert 701 may provide additionalcontext and insight as part of the automated translation process. Forexample, idiomatic and situational context related to specific contentmay be used in the training of the neural network and may result in amore natural and useful translation into one of many sign languages.

FIG. 7 illustrates, in part, the digitization of signing activity thatis received using a number of sensors 710 that can sense signingactivities of a user who uses sign language(s) (also referred to as anSL user 702). The captured data is then fed to one or more processors720 for processing. Due to the complexity of sign language, and in aneffort to support many sign languages, the amount of data that iscaptured may be prohibitive. Thus, embodiments of the disclosedtechnology may leverage data that has previously been captured anddigitized to reduce the amount of data that needs to be stored when thedevice is being used in real-time, either locally or in a remotesetting. The device then outputs textual or avatar rendering ofcommunication or content to the SL user via the front display 730 of thedevice.

The device can also include a rear display 740 to show textual or audiocommunication or content to a user that does not use sign languages(also referred to as a non-SL user 703). The device can receive standardaudio or textual communication from the non-SL user and may include arear control 750 for the non-SL user 703 to control the device.

In some embodiments, the device may be effectively used to perform signlanguage translations in a remote region, where access to studios and/ormore sophisticated computer technology is non-existent or very limited.In an example, a basic corpus of a sign language that is used in aremote area may be used to initially train the neural network and willallow translations upon arrival to that region. After the system isdeployed there, the corpus may be expanded exponentially based on inputby native sign language users, which will improve the translationcapabilities due to iterative training and interpretation (or execution)cycles of the neural network.

FIGS. 8A and 8B illustrate different views of an image capture andprocessing device that can be used for automated sign languagetranslation in accordance with an example embodiment of the disclosedtechnology. As illustrated in FIG. 8A, the image capture and processingdevice may include a right camera 810 and a left camera 850 to be ableto capture a moving object or scene (e.g., a sign language speaker) fromdifferent points of view, therein increasing the depth of fieldmeasurements that enable more accurate interpretation of the scene suchas the sign language gestures. Similarly, the inclusion of a rightmicrophone 820 and a left microphone 840 enable different contextual andenvironmental cues to be captured.

The image capture and processing device further comprises stereo (or 3D)camera 830, a front display 830, and one or more processors 870. In someembodiments, the one or more processors include an ARM Cortext-M3processor and at least one graphics processing unit (GPU). In otherembodiments, and as illustrated in FIG. 8B, the device may furthercomprise a rear display 880, which may be a touchscreen display. In someembodiments, the stereo camera 830 may be replaced or augmented by adepth sensor or multi-aperture camera, which may be configured tomeasure the “depth” or distance from the camera focal baseline to theobject corresponding to a particular pixel in the scene.

FIG. 9 illustrates an example flow diagram of operations that can becarried out by various component to implement automated sign languagetranslation in accordance with one or more embodiments of the disclosedtechnology. This example includes some features and components that aresimilar to those described above, and their description is not repeated.

As illustrated in FIG. 9 , multiple sensors 910 may each capture acommunication of a sign language user. In an example, using multiplesensors enables environmental factors to be acquired, and providesbetter depth of field measurements of sign language gestures. In someexemplary operations, a set of preprocessing operations can beperformed. For example, the input data collected from the multiplesensors is first aligned, both spatially and temporally. For example,based on the video quality and the external lighting and otherconditions, video conditioning procedures (e.g. color space conversion)may be implemented. This operation may be followed by spatial andtemporal filtering to, for example, reduce the data to a particularresolution, retain data for only a particular spatial zone of interestor a temporal period of interest. The processing may further include theapplication of image and/or video processing methods, e.g. edgedetection, which conditions the data for additional processing.

The conditioned data of the communication from the sign language usercan then be processed in order to extract features of gestures, facialcues and body cues, amongst other features that enable theidentification of the sign language. The input sign language istranslated to an internal representation, and subsequently translated tothe target language. The output is then rendered to the user.

In some embodiments, the feature extraction, identification andtranslation may be part of a neural network execution process. Beforethe neural network starts the execution process, the neural network istrained by the neural network learning process. The techniques discussedin later sections of this document can be implemented in the neuralnetwork learning process to allow the trained neural network torecognize a large number of characteristics in the input data moreefficiency and more accurately. To perform the neural network learningprocess, a set of training data can be used to carry out trainingalgorithms such as supervised training of the neural network. In someembodiments, as part of feedback for the learning process, thetranslated sign language is used to further train and modify the neuralnetwork to improve its identification and translation capabilities. Inyet other embodiments, reinforcement training of neural networks may beemployed to improve performance and increase the flexibility andadaptability of embodiments of the disclosed technology.

FIG. 10 illustrates a method 1000 that includes a set of operations thatcan be carried out to automate sign language translation in accordancewith an example embodiment of the disclosed technology. The method 1000includes, at operation 1010, receiving a digital representation of acommunication by a user in a first sign language. In some embodiments,the digital representation includes a plurality of images. In otherembodiments, the digital representation includes a video recording.

The method 1000 includes, at operation 1020, identifying the first signlanguage based on at least the set of gestures. In some embodiments,identifying the first sign language may be based on a sign languagegesture library or sign language content curated by an expert. In anexample, the expert content may comprise idiomatic and situationalcontext associated with the first sign language.

The method 1000 includes, at operation 1030, translating thecommunication in the first sign language, based on the identificationand the digital representation, to an internal representation. Themethod 1000 includes, at operation 1040, translating the internalrepresentation to at least one of a plurality of sign languagesdifferent from the first sign language. In some embodiments, thetranslation may be based on sign language content curated by an expert.For example, and when translating known subject matter (e.g. the Bible)the expert content may be based on existing interpretation and analysis.

In some embodiments, the method may further include receiving a responseto the communication, which is translated into the internalrepresentation, and subsequently into the first sign language.Embodiments of the disclosed technology are capable of real-timeoperation, which is enabled, in part, by the internal representation andthe underlying neural network.

As noted earlier, the example configurations in FIGS. 1-10 representexamples of systems that capture a variety of information (e.g., video,audio, still images, etc.) in different modalities (e.g., natural light,structured light, infrared light) of moving and still objects, as wellas of the background environment. As a result, a large amount of data isobtained to undergo further processing and analysis to extract theinformation of interest. Generation and analysis of large amounts ofdata are hallmarks of other systems and applications, such as autonomousvehicles and medical applications that involve analysis of medicalimages (e.g., MRI, X-ray, CT scan, video content, etc.). Additionalapplications include, but are not limited to, interactive video games,airport security and surveillance applications, analysis and trainingfor various sports, interactive home devices, and others.

In some embodiments, the example configurations in FIGS. 1-10 caninclude a device that supports multiple modalities in order to capturethe complexities and nuances of sign language for its communication andtranslation.

FIG. 11 illustrates an example system for sign language recognitionusing multiple input and output modalities. As illustrated therein, anumber of devices (Device 1, Device 2, . . . Device n, denoted 1112,1114, . . . 1118, respectively) each include multiple apertures (A1, . .. An) that are arranged around the subject 1120. The orientation of thedevices and apertures ensure that the nuances of movements of the signlanguage being communicated by the subject are captured. In an exemplaryimplementation, the multiple apertures are arranged so as to covermultiple angles (and perspectives) of the subject and in differentspatial planes. In other words, the multiple apertures are not allaligned on the same horizontal or vertical axis.

In an example, each of the devices (1112, 1114, 1116, 1118) illustratedin FIG. 11 typically use an approximately 90° horizontal field-of-view(HFOV), and they are generally oriented by less than half thefield-of-view in camera disparity applications (which refers to the useof multiple apertures to capture the same subject). Based on samplingand interferometric considerations, a system with three devices may havea first camera facing the subject head-on, a second camera 90° to oneside, and a third camera 45° to the other side. In one exemplary system,these three cameras may be placed in a single horizontal plane. Inanother exemplary system, the second or the third camera may bepositioned at an elevated position of 25-30° above the plane of theother two cameras. In yet another exemplary system, a fourth camera maybe placed at an elevated position with respect to the plane.

In some implementations of the disclosed technology, one or more of thedevices illustrated in FIG. 11 may be a special type of camera thatprojects a pattern of light (e.g., through a holographic diffuser) inthe Near IR region (˜850 nm, which is invisible to humans), and which isdetected by a silicon focal plane array (FPA). This advantageouslyenables depth information to be captured in higher detail as compared tousing pure stereoscopic imaging. This framework is typically referred toas a “structured light” camera. In this configuration, the projection“aperture” of the structured light can be mounted to a rigid structurewith the stereo apertures to ensure consistent alignment with thecameras and visible camera video can then be combined with the depthinformation.

By using multiple apertures (or equivalently, multiple input and outputmodalities) on a single device, and/or multiple multi-aperture devices,a more complete 3D model of a video scene can be captured in real timewith enough fidelity to enhance the performance of algorithms acting onthe data.

The use of multiple apertures results in the capturing of additionalinformation that cannot be not captured using existing technologies. Forexample, a conventional camera records light intensity from a singlepoint-of-view, and typically using a single aperture. In anotherexample, a light-field camera captures information about the light fieldemanating from a scene; e.g., the intensity of light in a scene, andalso the direction that the light rays are traveling in space.Light-field cameras are well-suited for static images (or scenes) andenable refocusing the image after the image has been taken.

In contrast to conventional technologies, implementations of thedisclosed technology capture an increased amount of information for thesame pixel using different cameras and apertures. For example, eachcamera of a plurality of cameras can capture a different view of thesame movement by the subject. While the disclosed devices can be readilyequipped with optical elements that can perform focusing in thetraditional sense, in one example, this increased amount of information(in an information theory sense) is captured without any refocusingconsiderations. As such, the captured data includes information that canbe used on an as-needed basis. For example, if there is a need toprovide a higher resolution image of a particular segment of thecaptured object, or to analyze a particular motion from differentangles, such information can be readily extracted from the captureddata. Further note that this increased amount of information becomesincreasingly more important to capturing motion, as compared to staticimages.

In some examples, each of the devices in FIG. 11 may be a single-FPAthat is capable of capturing depth information for an image or a frameof a video. Implementations of the disclosed technology may use imagingchips capable of sensing the phase angle of an incident ray on a singlepixel with no requirement of a corresponding multi-pixel lenslet. Inother words, the devices illustrated in FIG. 11 may effectively samplethe quadrature phase information of incoming electrometric radiation(e.g., light) thereby capturing depth information on a per-pixel basis.The use of one or more single- and multiple-aperture visual sensingdevices to capture a movement of a sign language from different anglesto subsequently enable robust and accurate identification of themovement is one of the capabilities of the disclosed technology.

As illustrated in FIG. 11 , the digital representation of the movementsof a sign language communicated by the user are transferred (e.g. usinga data transfer module 1140) to a processing module 1150. Someimplementations may include additional processing and/or hardwarecapabilities for pre-processing, time-aligning and post-processing thedifferent views of the subject, and subsequently interpreting them toidentify the movement communicated by the subject.

For example, each device illustrated in FIG. 11 may include an ARMprocessor running a variant of the Linux operating system, which may beused for the pre- and post-processing operations of the disclosedmethods. The pre- and post-processing operations may include filtering,transforming and other image processing operations. In someimplementations, the processing of the video and images through anartificial intelligence (AI)-based deep neural network (DNN) orconvolutional neural network (CNN) may be performed on-board, prior tothe off-platform transform.

More generally, numerous video processing operations, including but notlimited to timestamping, decoding/encoding, color space conversion,de-Bayering, and other signal and/or image processing, may be performedusing on-board GPU hardware in the device.

In an example, aligning the digital representations captured using themultiple apertures in the temporal domain may be implemented using a GPS(Global Positioning System) 1 PPS (pulse-per-second) signal or anetwork-based time service (e.g. NTP (Networking Time Protocol) or SMPTEtimecodes. In an example, the camera clocks may be synchronized usingNTP or the 1 PPS signal.

In other examples, the clocks for both the active and passive cameras inthe system are synchronized locally via a common clock signal based onthe support of the individual hardware devices. Some cameras maydirectly link their clock signals, but most commonly a frame integrationpulse is used, either rising and falling at the start of the frame, orstaying high through the integration duration, and then falling lowduring readout. Typically, the focal plane array (FPA) manufacturer(e.g., Sony, ON, Aptina, etc.) specifies the compatible pin signals forclock synchronization. This signal synchronizes the clocks locally, andcan then be synchronized globally either through the network, and/or via1 PPS or greater GPS sync lines from an on-board GPS receiver. Commonly,a GPS device is used in concert with the NTP software to providedistributed timing information to networked devices, which then “slew”their timing signal to match the reference, resulting in synchronizedframe captures throughout the networked devices. In some embodiments,the networked devices include multiple input modalities, e.g. adepth-field camera, a sound sensor and an infra-red (IR) camera. Forexample, the clocks in each of the modalities may be synchronized basedon the PPS or GPS signals.

In certain scenarios, the system may not necessarily require the datafrom all apertures to be registered or consolidated prior to processing.For example, the processing algorithms may process multiple camera feedsfrom multiple angles independently. While a single camera with oneaperture may be sufficient for the recognition of a simple signcorresponding to a single letter, e.g., “A,” a sign involving motionwould necessitate multiple apertures to be recognized accurately.Furthermore, reading the “emotion” of the subject may require facialanalysis from a completely independent data stream than the one used toidentify the sign language movements. Implementations of the system maybeneficially make the independent data streams available in both raw andprocessed formats, so that various (and very different) algorithms maybe used to robustly and accurately interpret sign language. In oneexample, the OpenPose library may be used to implement facialrecognition algorithms. In another example, algorithms that could beused for facial recognition may include principal component analysisusing eigenfaces, linear discriminant analysis, elastic bunch graphmatching using the Fisherface algorithm, the hidden Markov model, themultilinear subspace learning using tensor representation, and theneuronal motivated dynamic link matching. Thus, the accuracy of resultsmay be improved by including unique information, which is not possibleto observe except through implementations of the disclosed technology.

FIG. 12 illustrates another example system for sign language recognitionusing a device with multiple input and output modalities. As illustratedtherein, two sensors (Sensor 1 and Sensor 2 denoted 1212 and 1214,respectively) are arranged to ensure that nuances of sign languagemovements by the subject 1220 are captured in order to increase therecognition capabilities of the subsequent processing algorithms. In anexample, two sensors may be arranged with a 4° field-of-view (FOV)overlap. As illustrated in FIG. 12 , the two sensors are arranged toprovide a 45° FOV with an angular separation of (45−2×4)=37° angularseparation.

In alternate implementations, the desired FOV overlap may be computedfor multiple devices, and as discussed above, may be implemented suchthat the devices are not aligned along the same horizontal or verticalaxis. In general, the more the data collected by each sensing devicediffers, the richer the overall data set after processing will be.Furthermore, and in the context of being able to interpret the differentnuances of sign language (e.g., movement, emotion, etc.), the differentfeeds from each of the multiple sensing devices can be processed bydifferent algorithms. Having overlapped pixels (or more generally,information) from each device enables their alignment after possiblydisparate processing, and results in accurate and robust interpretationof signed language.

For example, one set of visual sensing devices can capture informationrelated to the movements of the sign language being performed by asubject's hands, which is processed by an AI-based DNN or CNN tointerpret its meaning. Additionally, a different set of visual sensingdevices can capture information related to the movement of the eyebrows,gaze direction and blinks of the subject, which is processed by facialrecognition algorithms. In an example, the subject may look upward whenindicating a hypothetical statement. Thus, implementations of thedisclosed technology are able to align and combine disparately processedresults to provide more accurate interpretations of the subject'sthoughts and meanings.

In some embodiments, the Sensor 1 and Sensor 2 may be implemented usingthe exemplary device illustrated in FIGS. 8A and 8B. Referring back toFIGS. 8A and 8B, the device illustrated therein may include a depthsensor that may advantageously augment the amount of informationcollection for signal language recognition, and which may be configuredto measure the “depth” or distance from the camera focal baseline to theobject corresponding to a particular pixel in the scene.

In a first example, the depth sensor may operate using structured lightprojections that are based using a light source to project a knownpattern, and using a receiver to detect the distortion of the reflectedpattern to calculate a depth map based on geometry. This approach canproduce very accurate depth measurement results, but can becomputationally expensive. Furthermore, structured light projections aresensitive to environmental brightness, and are typically used in dark orindoor areas.

In a second example, the depth sensor may operate based on thetime-of-flight (ToF) approach that relies on a light source to send outa pulse and a sensor to detect that pulse's reflection off the targetobject to record it's time of flight. The ToF-based depth sensor may beimplemented using a short pulse to provide very accurate (and moreexpensive) results, or it may use a modulated pulse and detect a phasechange, which provides less accurate (but much cheaper) results.

The use of a depth sensor (or equivalently, depth information obtainedfrom one or more apertures of one or more sensing devices)advantageously enables filtering out data that is not related to thesign (e.g., the gesture formed by the signer) itself. For example, thegesture/movement recognition algorithms can be used to remove unrelatedor unnecessary pixels from the image that are not within a desired depthrange. Additionally, a “3D” framework for the movement may beestablished using the depth information, so as to be able to detectnuances in not only the image plane, but even in planes that areperpendicular to the image plane.

FIGS. 13A, 13B and 13C illustrate another example device for signlanguage recognition using a device with multiple input and outputmodalities. As illustrated in FIG. 13A, the device includes active 3Dsensors 1315 and passive stereo sensors (1322 and 1324). As illustratedin FIGS. 13B and 13C, the example device 1310 from FIG. 13A may be usedas a handheld device when attached to a modular frame 1375 (as seen inFIG. 13B) in conjunction with a tablet 1380 (as seen in FIG. 13C),providing an implementation with input and output capabilities that issuited for sign language translation when more traditional (andexpensive) infrastructure is not available. The passive stereo sensors(1322 and 1324) are more economical than a full structured light sensor,but the latter provides an accuracy that may be two orders of magnitudegreater than that provided by the former.

For example, the device illustrated in FIGS. 13A-13C may be oriented sothe device 1310 is facing the signer, and the tablet 1380 is facing theperson holding the modular frame 1375. The device 1310, with the active3D sensors 1315 and the passive stereo sensors (1322 and 1324), cancapture the sign language movements communicated by the signer, performsign language recognition, and display a textual output of theinterpretation on the tablet 1380 screen. Alternatively, the tabletcould be facing the signer, the device (with a microphone) could befacing the person holding the modular frame. In this scenario, themicrophone can recognize speech, convert it to sign language, anddisplay it on the tablet using an avatar. Thus, a person is able tocommunicate with a signer using implementations of the disclosedtechnology.

The implementations illustrated in FIGS. 8 and 13 have multipleapertures that are closely co-located. This advantageously allows stereoprocessing, the ability to average out noise and improve signal-to-noiseratio (SNR), and enables using fewer devices. In one example, the deviceillustrated in FIGS. 13A and 13B may be a more complex version of thedevice illustrated in FIGS. 8A and 8B. For example, the devices in FIGS.13A and 13B may additionally include one or more of aspatial/DSP-processed mic array, a full structured light sensor and aUSB hub. Different example devices that are illustrated in variousfigures of this document provide improved sign language recognitioncapabilities using multiple apertures, and may be manufactured atdifferent price-points based on the additional capabilities supported.

As described above, using multiple apertures increases fidelity so as toenable the high-quality reproduction of the movement. This allowsadditional information for each pixel to be captured, which can be usedto create unique feature signatures for the different movements of thesign language. The features may be leveraged to identify the movementsin the subsequent processing stage. In an example, a feature signaturemay be the right hand of the subject moving horizontally within aparticular 3D volume in a particular amount of time. Features such asthese, in combination with other sign language movements and thesubject's emotions, may be mapped onto an interpretation of the signlanguage.

For example, the feature signatures from each of these differentmodalities may be combined through a point-cloud model, or amulti-camera, or multi-frame 3D model construction algorithms orartificial intelligence (e.g., DNNs, CNNs) programs, which enables moreaccurate and robust recognition. As expected, increasing the number offeature signatures used results in an increase in the training set aswell as the recognition network. In general, the moreunique/differentiated information is captured, the greater the accuracy(in statistical terms) of distinguishing one feature from another. Theuse of multiple apertures increases the amount of non-redundant datathat is captured by the system.

FIG. 14 illustrates example components of a system using a device forsign language recognition using a device with multiple input and outputmodalities. As illustrated in the example in FIG. 14 , multiple sensingdevices (denoted 1410, 1412, 1414, 1416 and 1418) may be connected to acommon processing structure that includes a GPU 1425, video processingcapabilities and data management capabilities (which may be, in anexample, co-located on a single CPU 1435), as well as communicationsupport (e.g., Wi-Fi 1447 and Ethernet 1449). The multiple apertures ofimplementations of the disclosed technology capture sign languagemovements from different angles, and may then use an artificialintelligence system 1465 for accurate and robust detection of themovements.

FIG. 15 illustrates a flowchart of an example method 1500 for signlanguage recognition using a device with multiple input and outputmodalities. The method 1500 includes, at operation 1510, capturing atleast one movement associated with the sign language using a set ofvisual sensing devices, the set of visual sensing devices comprisingmultiple apertures oriented with respect to the subject to receiveoptical signals corresponding to the at least one movement from multipleangles. In an example, the set of visual sensing devices comprises oneor more of an RGB color camera, a monochrome camera, a 3D stereo camera,a structured light emitter/receiver, or a time-of-flightemitter/receiver.

The method 1500 includes, at operation 1520, generating digitalinformation corresponding to the at least one movement based on theoptical signals from the multiple angles.

The method 1500 includes, at operation 1530, collecting depthinformation corresponding to the at least one movement in one or moreplanes perpendicular to an image plane captured by the set of visualsensing devices. In an example, collecting the depth informationincludes using a structured-light depth sensor or a time-of-flight depthsensor. In an example, the depth information includes a depth range fromone of the set of visual sensing devices to the subject's hands. Inanother example, producing the set of reduced information includesremoving at least some of the digital information that corresponds todepths not within the depth range.

The method 1500 includes, at operation 1540, producing a reduced set ofdigital information by removing at least some of the digital informationbased on the depth information.

The method 1500 includes, at operation 1550, generating a compositedigital representation by aligning at least a portion of the reduced setof digital information. In an example, the composite digitalrepresentation may be a point-cloud or a multi-frame three-dimensionalmodel. In another example, aligning at least the portion of the reducedset of digital information includes using one or more of a GlobalPositioning System (GPS) 1 pulse-per-second (PPS) signal, a networkingtime protocol (NTP) or an SMPTE timecode to temporally align part of thereduced set of digital information.

The method 1500 includes, at operation 1560, recognizing, using a neuralnetwork engine, the at least one movement based on the composite digitalrepresentation. In an example, recognizing the at least one movementbased on the composite digital representation uses an artificialintelligence (AI)-based deep neural network (DNN) and/or convolutionalneural network (CNN).

In some embodiments, the neural network engine may include one or moreconvolutional neural networks (CNNs) and one or more recurrent neuralnetworks (RNNs), which may be combined in architectures that allowreal-time processing for of the training images. A convolutional neuralnetwork (CNN or ConvNet) is a class of deep, feedforward artificialneural networks that typically use a variation of multilayer perceptronsdesigned to require minimal preprocessing. A perceptron is a computermodel or computerized machine devised to represent or simulate theability of the brain to recognize and discriminate. This means that thenetwork learns the filters (normally through a training process) neededto identify the features of interest; filters that in traditionalalgorithms were hand-engineered. This independence from prior knowledgeand human effort in feature design is a major advantage of CNNs. CNNshave been successfully used for image (or more generally, visual)recognition and classification (e.g., identifying faces, objects andtraffic signs) by using the “convolution” operator to extract featuresfrom the input image. Convolution preserves the spatial relationshipbetween pixels by learning image features using input (morespecifically, training) data.

In contrast to the CNN, a recurrent neural network (RNN) is a type ofartificial neural network where connections between nodes form adirected graph along a sequence. This allows it to exhibit dynamictemporal behavior for a time sequence. Unlike feedforward neuralnetworks, RNNs can use their internal state to process sequences ofinputs. That is, RNNs have a feedback loop connected to their pastdecisions, which lets the RNN exhibit memory. For example, sequentialinformation is preserved in the recurrent network's hidden state, whichmanages to span many time steps as it cascades forward to affect theprocessing of each new example. It is finding correlations betweenevents separated by many moments, and these correlations are called“long-term dependencies”, because an event downstream in time dependsupon, and is a function of, one or more events that came before.

The neural network engine takes the training image(s) and performs thetraining accordingly, e.g., using the CNN(s) and/or RNN(s). In someembodiments, the neural network engine executes on one or more graphicsprocessing units to leverage the parallel computing power. As discussedabove, the training process can be iterative—by evaluating theperformance and/or accuracy of the neural network process, the trainingsystem can determine if re-generating a different set of training imagesis necessary.

The method 1500 may further include capturing, using the set of visualsensing devices, one or more of an eyebrow movement, a gaze direction orone or more blinks of the subject that are associated with the at leastone movement, and where recognizing the at least one movement comprisesusing information associated with the captured eyebrow movement, thegaze direction or the one or more blinks to improve recognition of theat least one movement. In an example, the method 1500 may use facialrecognition algorithms to generate the information associated with theeyebrow movement, the gaze direction or the one or more blinks.

The method 1500 may further include capturing, using one or more audiosensors, an audible input associated with the at least one movement, andusing information associated with the audible input to improverecognition of the at least one movement. Subjects who primarily usesign language to communicate may try to accommodate for hearing peoplewith poor signing skills by vocalizing the words sometimes, or to conveyadditional aspects of the information being signed. Implementations ofthe disclosed technology are able to capture this audio input, and useit to improve the recognition of the movements of the signed language.

The method 1500 may further include capturing, using the set of visualsensing devices, external information indicated by the subject. Theexternal information, which may include the subject pointing to aportion of text, or an object or person in the vicinity of the subject,will typically augment the information being signed. This externalinformation can be captured and used to recognition of the associatedmovement.

FIG. 16 illustrates a flowchart of an example method 1600 for signlanguage recognition using a device with multiple input and outputmodalities. The method 1600 includes, at operation 1610, capturing atleast one hand gesture associated with a movement in the sign languageusing a set of visual sensing devices. In some embodiments, the set ofvisual sensing devices include multiple apertures oriented with respectto the subject to receive optical signals corresponding to the at leastone movement from multiple angles.

The method 1600 includes, at operation 1620, generating digitalinformation corresponding to the at least one hand gesture based on theoptical signals from the multiple angles. In some embodiments, themethod 1600 further includes the operation of combining the opticalsignals from the multiple angles after aligning their respectivetimestamps (e.g., using the 1 PPS or the GPS signal for synchronizationand alignment).

The method 1600 includes, at operation 1630, capturing at least oneenvironmental factor using a set of non-visual sensing devices.

The method 1600 includes, at operation 1640, combining the digitalinformation with information associated with the at least oneenvironmental factor to improve the recognition of the movement in thesign language.

FIG. 17 illustrates a flowchart of an example method 1700 for signlanguage recognition using a device with multiple input and outputmodalities. The method 1700 includes, at operation 1710, capturing atleast one movement associated with the sign language using a set ofvisual sensing devices that comprise multiple apertures oriented withrespect to the subject to receive optical signals corresponding to theat least one movement from multiple angles.

The method 1700 includes, at operation 1720, generating digitalinformation corresponding to the at least one movement based on theoptical signals from the multiple angles.

The method 1700 includes, at operation 1730, recognizing, using a neuralnetwork engine, the at least one movement based on the digitalinformation.

Some aspects of the disclosed embodiments relate to a non-transitorycomputer readable medium having processor code stored thereon includingprogram code for performing a method for recognizing a sign languagecommunicated by a subject. Such a method includes capturing at least onemovement associated with the sign language using a set of visual sensingdevices, where the set of visual sensing devices include multipleapertures oriented with respect to the subject to receive opticalsignals corresponding to the at least one movement from multiple angles.The method also includes generating digital information corresponding tothe at least one movement based on the optical signals from the multipleangles, collecting depth information corresponding to the at least onemovement in one or more planes perpendicular to an image plane capturedby the set of visual sensing devices, and producing a reduced set ofdigital information by removing at least some of the digital informationbased on the depth information. The method additionally includesgenerating a composite digital representation by aligning at least aportion of the reduced set of digital information, and recognizing,using a neural network engine, the at least one movement based on thecomposite digital representation.

In some embodiments, collecting the depth information includes using astructured-light depth sensor or a time-of-flight depth sensor. In someembodiments, the above noted method further includes capturing, usingthe set of visual sensing devices, one or more of an eyebrow movement, agaze direction or one or more blinks of the subject that are associatedwith the at least one movement. In such embodiments, recognizing the atleast one movement comprises using information associated with thecaptured eyebrow movement, the gaze direction or the one or more blinksto improve recognition of the at least one movement.

According to some embodiments, the above method further using a facialrecognition algorithm to generate the information associated with theeyebrow movement, the gaze direction or the one or more blinks. In yetanother embodiment, the set of visual sensing devices comprises one ormore of an RGB color camera, a monochrome camera, a 3D stereo camera, astructured light emitter/receiver, or a time-of-flight emitter/receiver.In still another embodiment, aligning the at least a portion of thereduced set of digital information includes using one or more of aGlobal Positioning System (GPS) 1 pulse-per-second (PPS) signal, anetworking time protocol (NTP) or an SMPTE timecode to temporally alignpart of the reduced set of digital information.

Implementations of the subject matter and the functional operationsdescribed in this patent document can be implemented in various systems,digital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.Implementations of at least some of the subject matter described in thisspecification can be implemented as one or more computer programproducts, e.g., one or more modules of computer program instructionsencoded on a tangible and non-transitory computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium can be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing unit” or “dataprocessing apparatus” encompasses all apparatus, devices, and machinesfor processing data, including by way of example a programmableprocessor, a computer, or multiple processors or computers. Theapparatus can include, in addition to hardware, code that creates anexecution environment for the computer program in question, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a stand-alone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of nonvolatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

While this patent document contains many specifics, these should not beconstrued as limitations on the scope of any invention or of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of particular inventions. Certain features thatare described in this patent document in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Moreover, the separation of various system components in theembodiments described in this patent document should not be understoodas requiring such separation in all embodiments.

Only a few implementations and examples are described and otherimplementations, enhancements and variations can be made based on whatis described and illustrated in this patent document.

1. An apparatus for recognizing a sign language communicated by asubject, comprising: plurality of sensors for capturing movement anddepth information corresponding to at least one movement associated withthe sign language; a processor; and a memory including instructionsstored thereupon, the instruction upon execution by the processor causethe processor to: remove at least some digital information correspondingto the at least one movement based on the depth information; generate acomposite digital representation based on the digital information thatis removed; and identify, using a neural network engine, the at leastone movement based on the composite digital representation.
 2. Theapparatus of claim 1, further comprising a depth sensor for collectingthe depth information, wherein the depth sensor is a structured-lightdepth sensor or a time-of-flight depth sensor.
 3. The apparatus of claim1, wherein at least some of the plurality of sensors are furtherconfigured to: capture one or more of an eyebrow movement, a gazedirection or one or more blinks of the subject that are associated withthe at least one movement, and wherein the instruction upon execution bythe processor further cause the processor to: identify the at least onemovement using information associated with the captured eyebrowmovement, the gaze direction or the one or more blinks.
 4. The apparatusof claim 3, wherein the processor is further configured to: use a facialrecognition algorithm to generate the information associated with theeyebrow movement, the gaze direction or the one or more blinks.
 5. Theapparatus of claim 1, wherein at least some of the plurality of sensorscomprise one or more of an RGB color camera, a monochrome camera, a 3Dstereo camera, a structured light emitter/receiver, or a time-of-flightemitter/receiver.
 6. The apparatus of claim 1, wherein the processor isfurther configured to receive one or more of a Global Positioning System(GPS) 1 pulse-per-second (PPS) signal, a networking time protocol (NTP)or an SMPTE timecode to temporally align part of the digitalinformation.
 7. The apparatus of claim 1, wherein the apparatus is ahandheld device that includes at least one display device, the at leastone display device allowing a sign language rendition of a capturedvoice, or a textual representation of a sign language gesture includingthe at least one movement.
 8. The apparatus of claim 1, wherein theapparatus is implemented as a multi-component device, and wherein the atleast some of the plurality of sensors include three visual sensingdevices, a first of the three visual sensing devices is positioneddirectly in front of the subject, a second of the three visual sensingdevices is positioned substantially 90° to one side of the subject, anda third of the three visual sensing devices is positioned substantially45° to another side of the subject.
 9. The apparatus of claim 8, whereinthe first, second and third of the three visual sensing devices are in asingle horizontal spatial plane.
 10. The apparatus of claim 8, whereinthe first and second of the three visual sensing devices are in a singlehorizontal spatial plane, and wherein the third of the three visualsensing devices is in an elevated position approximately 25° to 30°above the single horizontal spatial plane.
 11. The apparatus of claim 1,wherein at least some of the plurality of sensors further captureexternal information indicated by the subject, wherein the externalinformation is associated with the at least one movement, and whereinidentification of the at least one movement comprises using the externalinformation to improve identification of the at least one movement. 12.The apparatus of claim 11, wherein the external information comprises aportion of a text, an object in a vicinity of the subject, or anotherperson.
 13. The apparatus of claim 11, wherein the external informationcomprises environmental factors.
 14. A method for recognizing a signlanguage communicated by a subject, comprising: capturing movement anddepth information corresponding to at least one movement associated withthe sign language; producing a reduced set of digital information byremoving at least digital information corresponding to the at least onemovement based on the depth information; generating a composite digitalrepresentation based on the digital information that is removed; andidentifying, using a neural network engine, the at least one movementbased on the composite digital representation.
 15. The method of claim14, wherein the depth information is captured using a structured-lightdepth sensor or a time-of-flight depth sensor.
 16. The method of claim14, further comprising: capturing one or more of an eyebrow movement, agaze direction or one or more blinks of the subject that are associatedwith the at least one movement, and wherein identifying the at least onemovement comprises using information associated with the capturedeyebrow movement, the gaze direction or the one or more blinks toimprove recognition of the at least one movement.
 17. The method ofclaim 16, further comprising: using a facial recognition algorithm togenerate the information associated with the eyebrow movement, the gazedirection or the one or more blinks.
 18. (canceled)
 19. The method ofclaim 14, wherein generating the composite digital representation basedon the digital information that is removed includes using one or more ofa Global Positioning System (GPS) 1 pulse-per-second (PPS) signal, anetworking time protocol (NTP) or an SMPTE timecode to temporally alignpart of the reduced set of digital information.
 20. The method of claim14, wherein the composite digital representation comprises a point-cloudor a multi-frame three-dimensional model.